We can begin to find our answer about whether Jesus had literal brothers by looking at the accounts of his family trying to talk to him. The scene in question begins with self-righteous religious leaders demanding that he give them a sign to prove he was the Messiah (Matthew 12:38). Jesus rightfully labels this request as evil and rejects it. He then states that the only sign that will be given to reveal his spiritual identity will be his resurrection from the dead after three days.
After Jesus further chides the request for a sign from a "wicked and adulterous generation" (verse 39), his mother Mary and his brothers desire to speak with him. Three of the four Gospel accounts record their attempt to get someone near Jesus to get his attention and let him know what they want.
But while He (Jesus) was still speaking to the multitudes, behold, His mother and His brothers were standing outside, seeking to speak with Him (Matthew 12:46, HBFV, see also Mark 3:31, Luke 8:19).
Who is my family?
In a response the multitude could hear, Jesus rhetorically asks who is his mother and brothers (Matthew 12:48). He then answers his own question by stating he does not consider his most close relationships (as do most humans) to be limited to only those who have some physical tie to him. In contrast, he states he is willing to have a close spiritual relationship, like a family, with anyone (e.g. his disciples, verse 49) who does the will of (obeys) God the Father (verses 48 - 50).
The Lord's response, spoken to a large crowd of people eager to be with him, would make no sense if the initial request to talk with him was made by anyone who was not physically related to him.
Names
Surprisingly, in Matthew 13, the Bible reveals the names of four of Christ's physical brothers. It also reveals he had more than one sister! This information is revealed to us in the comments made by those in Nazareth, who knew Jesus and his family for many years, in response to his profound and unique teachings at a local synagogue.
From where did this Man (Jesus) receive this wisdom and these works of power? Is this not the carpenter's son? Is not His mother called Mary, and His brothers James and Joses and Simon and Judas? And His sisters, are they not all with us? . . . (Matthew 13:55 - 56).
We find additional evidence that Jesus did have brothers in the Book of Acts. His remaining eleven disciples he personally taught (Judas had committed suicide), after witnessing his ascension to the Father (Acts 1:4 - 9), went back to Jerusalem. While they waited in the city for the promised Holy Spirit (verses 4 - 5), they gathered to pray together with the following people.
All these were steadfastly continuing with one accord in prayer and supplications, together with the women, including Mary, the mother of Jesus, and with his brothers (Acts 1:14).
The Bible reveals that Jesus did have (half) brothers and (half) sisters (they all had the same mother, but not the same father). Mary, by the time her firstborn son's ministry started, had given birth through Joseph to at least four other boys and at least two girls for a family composed of at least seven children!